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(54) Camera control 

(57) An image received by the camera is stored in 
a memory (24) and analysed (25) to obtain for each pic- 
ture element of the image a score indicating the degree 
of dissimilarity of the element and its environment from 
the remainder of the image. Thee scores are further an- 



alysed (27) to identify a region of the image having the 
greatest such dissimilarity, and the coordinates at this 
point are used to direct a focus control mechanism (22) 
to focus the camera on that part of the image, and/or to 
direct an exposure control device (24) to adjust the ex- 
posure of a camera to suit the identified part of image. 
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Description 

[0001] This invention relates to a system for camera 
control and particularly, though not exclusively for auto- 
matically focusing camera lenses on the subject in a 5 
scene. The system may also be used to ensure accurate 
exposure of the subject in a scene and to offer guidance 
to the photographer on composition. 
[0002] Modem cameras employ either active or pas- 
sive autofocus systems. Active systems use sound 
waves or infrared to judge the distance of subjects from 
the camera and hence derive the necessary lens adjust- 
ment to obtain a sharp image. Early cameras emitted 
an ultra-high-frequency sound and then listened for the 
echo; the length of time it took the ultrasonic sound to 
reflect from the subject determined the focus setting. 
[0003] Systems that use sound are very limited and 
cannot be used to shoot through glass, for example. 
Systems that use infrared (US Patents 4602861, 
4983033, 4843416) can measure aspects (e.g. intensity 
or the delay) of the reflected beam to estimate the dis- 
tance of the subject from the camera. Such systems are 
misled by a source of infrared from an open flame, or a 
black object that absorbs the outgoing infrared beam. 
As with sound the infrared beam can bounce off objects 
(e.g. fence, bars, glass, etc) in front of the subject there- 
by producing erroneous distance measures. In addition 
very bright subjects can make it difficult for the camera 
to 'see' the reflected infrared beam. 
[0004] Passive autofocus determines the distance to 
the subject by computer analysis of the image captured 
by the camera lens. A common passive technique 
measures the sum of the differences in intensity of light 
arriving on adjacent pixels in a horizontal and/or vertical 
strip and adjusts the focus until a maximum sum is ob- 
tained. There is no limitation on the distance to the sub- 
ject and focusing can take place through glass. 
[0005] Both autofocusing systems either assume by 
default that the subject is in the centre of the image or 
rely upon the photographer to identify the point of focus 
to the camera. This means that it is quite easy to mis- 
takenly focus on the background if the actual subject is 
off-centre, or if the camera is accidentally focussed on 
a point that misses the subject. Some systems apply 
special processing rules to high-lights and dark areas to 
produce better results. However, these rules cause 
many focusing failures on images to which the rules do 
not apply. 

[0006] According to the present invention there is pro- 
vided a camera comprising means for storing electron- 
ically an image received by the camera; 

means for analysing the stored image to identify 
a region of the image different from other parts of the 
image; and 

means for controlling operation of a camera based 
on the identified part of the imatge. 
[0007] Other aspects of the invention are defined in 
the subclaims. 



[0008] Some embodiments of the invention will now 
be described, by way of example, with reference to the 
accompanying drawings in which: 

Figure 1 is a block diagram of a camera in accord- 
ance with one embodiment of the invention; 

Figure 2 is a flowchart illustrating the operation of 
the camera; 

Figure 3 is a flowchart illustrating the analysis of a 
still picture; and 

Figure 4 represents a still picture to be processed 
by the method of Figure 1 . 

[0009] Figure 1 shows a camera having a lens 20 
which focuses the image of a subject on a digital pick- 
up device 21 . The focusing of the image is controlled by 
a focus device 22 which adjusts the focus on the basis 
of feedback from the pick-up device 21 , on the basis of 
an input signal stipulating (for example by means of x, 
y coordinates) the location within the field of view of the 
camera upon which the focusing operation is to take 
place. The selection of this point can be set by means 
of a manual control 23. The camera also has an expo- 
sure control device 24. 

[001 0] As so far described, the camera is convention- 
al. For the purpose of this description it is assumed to 
be a still, digital camera. However it could equally well 
be a digital video camera, or camcorder for recording 
moving pictures. Indeed it could be a camera whose pri- 
mary task is to record pictu res in a more traditional man- 
ner, for example upon a photographic film (still or cine) 
or a television camera employing a non-digital pick-up 
device. In the latter cases, the digital image pick-up de- 
vice 21 would be additional to the components that such 
a camera would normally contain. 
[0011] The components illustrated in Figure 1 further 
comprise a digital image memory 24 into which image 
pixel values from the pick-up device are stored, a Visual 
Attention Map processor 25 which computes a set of vis- 
ual attention values for pixels in the image memory 24 
and places them in the Visual Attention Map memory 
26, and a subject location processor 27 that determines 
the location of the subject from data in the map 26 and 
communicates this location to the autofocus mechanism 
22. 

[001 2] In another arrangement the autofocus mecha- 
nism may not directly affect the focus.but merely display 
the indicated subject location in the viewfinder 28 for the 
photographer to use in the composition of the picture. 
The photographer may then choose whether the focus- 
ing operation is to take place on the recommended lo- 
cation, or the centre of the scene, or on a manually se- 
lected point. 

[0013] A flowchart indicating the operation of the proc- 
ess is shown in Figure 2. An image is captured (step 1 ) 



15 



20 



25 



30 



35 



40 



45 



50 



EP 1 286 539 A1 



3 

through the camera lens system and stored in the mem- 
ory 24. In step 2 the processor 25 generates a Visual 
Attention Map 26 consisting of a value (typically from 1 
to 100) for each pixel of the image, which a high value 
corresponds to a high likelihood that the corresponding 5 
pixel in the image memory 24 lies on the principal sub- 
ject. The processor 27 in step 3 calculates the x, y co- 
ordinate of the subject in the captured image 24 using 
data in the store 26. 

[0014] The operation of the Visual Attention Map 10 
processor 25 will now be described. This method is de- 
scribed and claimed in ourco-pending international pat- 
ent application no. PCT/GB01/00504. 
[0015] Considering now Figures 3 and 4, an image to 
be analysed is stored in a digital form in the image store 15 
24, as an array A of pixels x where each pixel has colour 
intensities (r x , g x , b x ) attributed to it. The procedure fol- 
lowed by the processor 25, implemented by a micro- 
processor controlled by suitable software, is as follows. 
[0016] A pixel x 0 is then selected from the array A 20 
(step 1), and its intensity value (r x , g x , b x ) is stored in a 
test pixel register (not shown). 
[0017] An anomaly count c x , and a count of the 
number of pixel comparisons l x are both set to zero (step 
2). 25 
[0018] The next step is the random selection of a 
number of points in the vicinity of the test pixel x 0 . This 
region is defined by a distance measure u x (typically in 
units of pixels). Thus, n pixels Xj are selected such that 

dist(Xj - Xj^) < u x 30 

where j = 1, n and x 0 = x. 
[0019] The distance used may be any of those con- 
ventionally used, such as the Euclidean distance or the 
"city block distance between the positions within the im- 
age of the two pixels. If the horizontal and vertical coor- 35 
dinates of Xj are p(Xj) and q(xp then the Euclidean dis- 
tance is 

>(*;)-/>(* y -,)f +[<?(*,)- ?(*/-.)f 40 
whilst the city block distance is 

[0020] Typically n = 3 , and u x = 1 . An example of such 
a group is shown in Figure 4, in which the test pixel, 
(shown boxed) has pixels (shown shaded) associated 50 
with it. For u x = 1 , the pixels are contiguous, but, in gen- 
eral the pixels may not necessarily neighbour one an- 
other or be contiguous in any sense. The definition of 
the neighbour pixels is stored in a neighbour group def- 
inition store. 55 
[0021] A pixel y 0 is now selected randomly (step 6) 
from the array A to be the current comparison pixel (also 
shown boxed in Figure 3) whose identity is stored in a 



comparison pixel register. 

[0022] The value of l x stored in the comparison coun- 
ter is incremented (step 7): if a limit L is exceeded, no 
further comparisons for the test pixel x are made (step 
8). The contents of the neighbour group definition reg- 
ister are then used by the calculation processor 25 to 
define a set of pixels forming a test group Xj and a set 
of pixels forming a comparison group yj, each pixel yj of 
the comparison group having the same positional rela- 
tionship to the comparison pixel y as the corresponding 
pixel ^ in the test group has to the test pixel x (step 9). 
[0023] The processor 25 then compares each of the 
pixels Xj (shaded in Figure 3) with the corresponding pix- 
el yj (also shown shaded), using a set of threshold val- 
ues Ar x , Ag x and Ab x . 

[0024] A pixel y is identified as being similar to a test 
pixel x if : 

I r y - r x| < *r x 

and 

|9y " 9x| < A 9x 

and 

IV b x| <Ab x- 

- where Ar x , Ag x and Ab x are threshold values which 
are, in this embodiment, fixed. 

[0025] If all the pixels Xj in the test group are similar 
to their corresponding pixels yj in the comparison group, 
the process is repeated by selecting a new set of neigh- 
bouring pixels (step 5) and a new comparison pixel y 0 
(step 6). If (as illustrated in Figure 4) one or more pixels 
Xj in the test group are not similar to the corresponding 
pixel ^ in the comparison group, in accordance with the 
similarity definition above, the count c x is incremented 
(step 10). Another comparison pixel y Q is randomly se- 
lected and stored in the comparison pixel register (return 
to step 6) and the neighbour group definition retrieved 
from the neighbour group definition store is used to sup- 
ply a new comparison neighbour group to the compari- 
son group register 449 for comparison with the test 
group stored in the test group register. A set of pixels Xj 
is retained in the test group register so long as it contin- 
ues to fail to match other parts of the image. Such a set 
represents a distinguishing feature of the locality of x - 
the more failures to match that occur, the more distinc- 
tive it is. The more comparison pixels y that the test pixel 
x fails to provide matches for, the higher the anomaly 
value c x stored in the anomaly counter becomes. Con- 
versely, the more matches that the test pixel x gener- 
ates, the lower the value of the anomaly value when the 
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threshold L is reached by the comparison counter. As L 
comparisons are made each time, the anomaly value c x 
which results from the process may be considered to be 
a measure of the proportion of randomly selected pixels 
which would fail to provide a match for the test pixel x. 
[0026] When the iteration value l x stored in the com- 
parison counter reaches the threshold value L, the iter- 
ative process stops (step 8) and the current anomaly 
value c x is output as the anomaly value or score for the 
pixel x. This final anomaly value c x is the measure of 
visual attention for the test pixel x, and is the number of 
attempts (from a total of L attempts) for which the inher- 
ent characteristics (i.e. the colours) of randomly select- 
ed neighbours of pixel x failed to match the correspond- 
ing neighbours of randomly selected pixels y. A high val- 
ue for c x indicates a high degree of mismatch for pixel 
x with the rest of the image and consequently that pixel 
x is part of an object worthy of visual attention. 
[0027] In this embodiment, the process is repeated, 
from step 1 , for every pixel in the image as the test pixel, 
so that a value c x is obtained for every pixel x in the array 
A. Typically, L may be set to be 100. 
[0028] As described above, comparisons are per- 
formed for the neighbouring pixels Xj,y |f j=i f ...n; however, 
if desired, the original or root pixels may also be includ- 
ed, the comparisons being performed for j = 0, ...,n. 
[0029] Some modifications will now be discussed: 

1 , The thresholds used in an embodiment of the in- 
vention for colour images will depend upon the col- 

- our space in which the comparison between pixels 
is carried out. In another embodiment of the inven- 
tion operating on colour images in the hue, satura- 
tion, value (HSV) space Ah x , As x , Av x colour differ- 
ence thresholds can be used. Here, pixel values 
consist of h x , s x and v x and the calculation is carried 
out in the HSV colour space pixel y is identified as 
being similar to test pixel x is: 

|v y - vj < Av x , and |s y - s x | < As x , and |h y - h x | < Ah x 

where Ah x = Z*(2- v x )*(2- s x ). Z is stored in an 
empirical table of thresholds dependent upon h x . 
This results in a larger value of Ah x for low values 
of v x and s^. 

Alternatively, for a grey-scale image, single lu- 
minance values t x and a luminance difference 
threshold ^ would be used. 

2. The selection of a group of n pixels Xj in the neigh- 
bourhood of the test pixel x from the image store 24 
may be made such that: ' 

dist(x j ,x (j . 1) )<u x 

where] = 1, .... n and Xq = x 



If desired, u x may vary with j: this allows pixels 
to be selected from a wide region whilst ensuring 
that some of the selected pixels are close to the test 
pixel x 0 . The value of dist (Xj, x^) may be defined 
5 in any suitable units, such as pixel size. The defini- 
tion of the neighbour group is stored in the neigh- 
bour group definition store 444. 

Alternatively, a group of n pixels Xj in the neigh- 
bourhood of the test pixel x can be selected from 
10 the image store 440 , the selection being such that: 

dist(x 0 , x g) ) < u x 

15 where j = 1 , n and Xq = x 

As in the case of the still image, the distance 
function used can be any of those conventionally 
employed. 

20 3. In the first embodiment of the invention the colour 
difference thresholds were predetermined and were 
not changed with each selection of a new neighbour 
group definition strategy. Alternatively the search 
strategy selected by the CPU 42 and provided to a 

25 neighbour group definition store may comprise a set 
of colour difference thresholds (Ar x , Ag x , Ab x ), (or in 
the case of grey level images a single threshold Atj), 
as well as the neighbour group definition. Previous- 
ly generated search strategies, comprising neigh- 

30 bour pixel groups definitions Xj and associated col- 
our difference thresholds (Ar x , Ag x , Ab x ) stored in 
the search strategy store as a result of achieving a 
high anomaly score on previous test pixels may be 
preferentially selected by the CPU 42, randomly 

35 generated candidates only being supplied by the 
processor to the current neighbour group definition 
store when the supply of such stored criteria is ex- 
hausted. This mechanism reduces the number of 
unsuccessful iterations of the process and enhanc- 

40 es the anomaly values in the vicinity of the object of 
attention by reusing features that highlight mis- 
matches in the current image. 

Similarly, test groups that have achieved high 
anomaly scores on previous tests may be retrieved 

45 from the search strategy store. 

As the process continues, successful search 
strategies (that is, combinations of values of Ar x , 
Ag x , Ab x and u x , and neighbour groups, which gen- 
erate high values of c x ,) will become apparent. If a 

50 group of n pixels Xj and the corresponding colour 
difference thresholds (Ar x , Ag x , Ab x ) cause the 
anomaly value of c x to reach a threshold M before 
a match is found, the search strategy stored in the 
neighbour group definition store is copied to the 

55 search strategy store for future use, if it is not al- 
ready stored. The strategies that have generated 
high anomaly values are thus available in the 
search strategy store for use in selecting suitable 
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values in further cycles. Once a match is found, the 
process starts again with a new search strategy 
(colour difference threshold and neighbour set) 
stored in the neighbour group definition store, either 
by retrieval from the search strategy store or gen- 
erated randomly. 

Initially the search strategies will be generated 
at random by the processor, — if the strategy is not 
suitable for identifying differences the cycle will be 
rejected and a new strategy selected. Successful 
strategies can be stored in the search strategy store 
for subsequent re-use. 

5. Several test pixels may be processed in parallel, 
but for purposes of illustration only one will be con- 
sidered here. 

6. In the above examples, an anomaly value c x is 
formed for every pixel of the array. However, in order 
to increase the speed of operation, the values c x 
may be formed only for a subset of the pixels. Once 
attention values have been generated for the pixels 
in the subset, then further pixels in the vicinity of 
those having a high measure c x may then be proc- 
essed. For example one might choose the top 20% 
of the pixels (in terms of measures c x ) and process 
the pixels within a small defined area of each. 

[0030] In the illustrated embodiment the pixels form a 
regular rectilinear tessellation, but the process is suita- 
ble for other arrangements of pixels. If the array is irreg- 
ular, the positional relationship of each pixel yj to the 
comparison pixel y may not be exactly the same the po- 
sitional relationship of each pixel Xj to the test pixel x, 
but each one will be the closest possible to the exactly 
corresponding position. 

[0031] Turning now to the detailed operation of the 
subject location processor 27, one method of proceed- 
ing is as follows. For example, the values in the store 
26 are first set to zero if they are less than 90% of the 
highest value. If either the width or height of the remain- 
ing non-zero array is greater than half the width or 
height, respectively, of the image stored in 26, the sub- 
ject location has not been found (step 4) and the auto- 
focus point is set to the centre of the image (step 6). If 
this is not the case, the centre of gravity of the array is 
computed. If this co-ordinate corresponds to a non-zero 
value in the Visual Attention Map 43, it is set as the x t y 
co-ordinate of the autofocus point (step 5). Otherwise 
the x,y co-ordinate of the closest non-zero value in the 
memory 26 is taken as the autofocus point (step 5). The 
location of the autofocus point is then displayed in the 
viewfinder (Step 7) and the autofocus mechanism set to 
operate at the selected autofocus point (step 8). 
[0032] At any time during this process a manual over- 
ride (step 9) is available which forces the autofocus 
mechanism to operate on a point in the image deter- 
mined by the user. It is also the case that as the camera 



is moved, a series of images is being captured into im- 
age memory 24 and new autofocus points are continu- 
ously being generated and displayed. 
[0033] The system described here is passive and first 

5 identifies the location of the principal subject in a visual 
scene and directs a computer analysis in the neighbour- 
hood of that location to determine the distance of the 
subject from the camera. The subject is located using 
an algorithm that highlights anomalous parts of the im- 

10 age that bear little similarity with other parts of the im- 
age. In the process a Visual attention Map is produced 
that associates a score. The system is not affected by 
lighting conditions providing sufficient light enters the 
camera from the subject. Neither is the system affected 

15 by camera movement because the method of subject 
identification is independent of the position of the sub- 
ject within the scene. The invention also provides the 
benefit of focussing upon small or narrow subjects that 
are at different distances from the camera to the back- 

20 ground. For example, this is the case for an image of a 
small flower against a distant background, or a distant 
object viewed through a small hole in a wall that is close 
to the camera. The autofocus mechanism is also appli- 
cable to video cameras where each frame is analysed 

25 in a similar fashion and the camera lens is adjusted ac- 
cording as the identified subject moves closer or further 
away from the camera. 

[0034] The method may also be applied to automatic 
exposure control in addition to, or instead of, focus con- 
30 trol. It is common for most cameras to rely upon a centre- 
weighted exposure metering system which provides an 
average exposu re value for the whole picture, but based 
principally upon the content at the centre of the scene. 
However, if the real subject is off-centre, or if the back- 
35 ground still has an undue effect on the exposure value 
(e.g. seagull in a bright sky), results can be poor. As with 
autofocusing, this invention enables an exposure value 
to be determined only from light reflected from the sur- 
face of the subject in a scene thereby ensuring that the 
40 subject is represented faithfully in the final photograph. 
Thus the coordinates from the subject location proces- 
sor 27 are fed to the exposure control unit 24 in order to 
cause the exposure control unit (which otherwise oper- 
ates in conventional manner) to take its exposure read- 
45 ings from the identified part of the image. 

[0035] In a more advanced version of the invention for 
a digital camera the exposure may be varied across the 
image according to the locations of the principal sub- 
jects in the scene. This would mean that the photogra- 
50 pher would not only have the guarantee of correct ex- 
posure for the subject, but he/she would also have the 
additional facility to correct the exposure for the back- 
ground. In the case of a subject in shade against a sunny 
background the exposure for the background would be 
55 treated independently of the subject and would not be 
overexposed. In a similar fashion the shaded back- 
ground to a subject in bright sunlight would not be un- 
derexposed. 
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[0036] In another arrangement for a digital camera, 
the autoexposure mechanism may not directly affect the 
exposure used to take the photograph, but instead 
cause the potential image to be displayed in the view- 
finder for the photographer to consider during composi- 
tion. The effects of variable exposure if applicable would 
also be displayed. 

[0037] The method described in this invention may al- 
so be applied to the autoexposure process for image 
scanners. Exposure estimation is normally based upon 
a prescan of the entire image from which an average 
exposure level is calculated. This exposure level takes 
no account of the subject content and often attaches too 
much weight to the background (e.g. sky) in achieving 
a full tonal range. This invention attaches a greater 
weight to the subject during the calculation of the expo- 
sure and thereby produces a better representation of the 
subject in the scanned image. 
[0038] It is of further benefit that the information 
gleaned to identify the subject in a scene may also be 
used to achieve very high storage compression for the 
image without reducing perceptual quality. Moreover 
this information will also be available to automatically 
crop the image to remove areas of background, either 
within the camera (to reduce storage demands still fur- 
ther) or subsequently in an image editing software pack- 
age. 
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es or does not match the comparison group; 
selecting further said comparison groups and 
comparing them with the test group; 
generating a distinctiveness measure as a 
5 function of the number of comparisons for 

which the comparison indicates a mismatch. 

3. A camera according to Claim 1 or 2 in which the 
control means is operable to adjust the focus of the 

10 camera such that the identified part of the image 
shall be in focus. 

4. A camera according to any one of the preceding 
claims in which the control means is operable to ad- 

15 just the camera exposure on the basis of the light 
output of the identified part of the image. 

5. A camera according to any one of the preceding 
claims including means operable to display in the 

20 viewfinder of the camera an indication as to the po- 
sition of the identified location within the image. 

6. A camera according to Claim 5 in which the expo- 
sure control means is operable to apply different ex- 

25 posures to different parts of the image. 



Claims 30 

1. A camera comprising means for storing electroni- 
cally an image received by the camera; 

means for analysing the stored image to iden- 
tify a region of the image different from other parts 35 
of the image; and 

means for controlling operation of a camera 
based on the identified part of the image. 

2. A camera according to Claim 1 , in which the analy- 40 
sis means comprises processing means arranged 

in operation to: 

select from the stored image a group of test pic- 
ture elements comprising at least two ele- 
ments; 

select a group of comparison picture elements 
comprising at least two elements, wherein the 
comparison group has the same number of el- 
ements as the test group and wherein the ele- 50 
ments of the comparison group have relative to 
one anotherthe same positions in the image as 
have the elements of the test group; 
comparing the value of each element of the test 
group with the value of the correspondingly po- 55 
sitioned element of the comparison group in ac- 
cordance with a predetermined match criterion 
to produce a decision that the test group match- 
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